The present invention relates generally to compression of in-memory data stores, and more particularly, but not by way of limitation, to a system, a method, and a recording medium for reducing memory consumption for in-memory data stores.
Conventional in-memory, key-value pair systems consume memory from indexing data structures in which, for each key-value pair, an index entry is created, and it requires a space within the indexing structure to store each entry. Memory is consumed from the memory store for each of the key-value items in which the conventional systems store received data as it is within the storage area without resorting to a compression mechanism.
Other conventional in-memory, key-value systems aim to combine data deduplication with compression to reduce the storage footprint for the file systems. However, such systems require a two-step method to reduce storage consumptions in which a file must be partitioned into multiple segments and then many restrictions must be followed to choose which segments would be compressed together.
Other conventional in-memory, key-value systems merely include a database in which the systems aim to reduce the number of columns within the database by finding and eliminating functionally dependent columns. However, such conventional systems are limited by the similarity detected since the system is based on the assumption that rows must have a certain extent of similarity such that a single row can be used as the representative row while the others can be removed.
However, each conventional in-memory, key-value system above, and all other conventional in-memory key-value systems are limited in their application in that they do not consider packing, via a server, data according to certain conditions and compressing continuous key-value pairs to generate blocks, managing the primary indexing structure of the blocks instead of entries for each of the individual key-value pairs, and offloading the block decompression to a front-end server such that the back-end server can return the block that contains the requested key without decompression.
That is, there is a technical problem in the conventional systems that indexing data by conducting an exact lookup increases the size of indexing data structures (i.e., individual key-value pairs) and storing received data in the form in which it is received (i.e., each key-value pair individually) which consumes a lot of metadata.